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Abstract Body 


Background / Context: 

Nearly one-quarter of U.S. eighth graders score below basic on national assessments of reading 
(NCES, 2013) and are poorly equipped for the reading demands of secondary school. Struggling 
adolescent readers cannot summarize a simple passage, use context to determine word meanings, 
and have difficulties making text-based inferences. In addition, poor fluency limits many 
struggling readers’ ability to process text efficiently, compromising basic and inferential 
comprehension (Cantrell et ah, 2013; Flynn et ah, 2012). 

Purpose / Objective / Research Question / Focus of Study: 

In this study, we present intention-to-treat impacts from a randomized clinical trial of the 
Strategic Adolescent Reading Intervention (STARI) on a scenarios-based assessment of reading 
comprehension. STARI is a multicomponent reading intervention for struggling middle school 
readers and is implemented as a year-long supplemental reading program for middle school 
students who score below proficient on state literacy assessments. Building from a multiple- 
component view of reading development, STARI incorporates strands on decoding, fluency, 
vocabulary, and comprehension, and provides multiple supports for struggling adolescent 
readers. For example, teachers provide explicit instruction on strategies for decoding 
multisyllabic words, focused on identifying more complex letter combinations, syllable patterns, 
and morphological units such as base words and affixes. Instructional activities that target these 
skills have shown success in improving older struggling readers’ word recognition and fluency 
(Edmonds et al., 2009). In previous work, we found positive intention-to-treat effects on 
multiple domains of reading, including word reading, morphological awareness, and efficiency 
of basic reading. This study extends prior work by examining effects on a Global, Integrated 
Scenario-Based Assessment (GISA) that is designed to assess a broader conception of reading 
ability (Sabatini et al., 2014). 

Setting: 

Four Massachusetts school districts served as research sites: two large urban districts, District A 
and B, and two rural/suburban districts, District C and D. Districts volunteered to be part of the 
study and solicited schools to participate (in the case of the larger districts) or had all their 
middle schools participate in the two smaller districts. 

Population / Participants / Subjects: 

A total of 554 students in grades 6, 7, and 8 were included in the final analytic sample. On 
average, the demographic characteristics of the schools included mostly low-income children 
(68% of children were eligible for free or reduced lunch across study schools). 

Intervention / Program / Practice: 
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This experimental study was implemented in the 2013-14 school year as part of the Reading for 
Understanding (Rfu) initiative, which supports the development and evaluation of research-based 
reading comprehension interventions from the elementary to secondary grades. In response to 
the challenge of accelerating multiple domains of reading for struggling adolescents, we 
developed two components of START (a) an intensive graduate course and on-site coaching for 
participating teachers and (b) a targeted intervention for students whose reading skills were 
approximately two levels below grade norms. The STARI intervention provided students with 
an average of 60 additional minutes of targeted intervention per session, supplementing the 
regular English language arts curriculum. 

The Strategic Adolescent Reading Intervention (STARI) is a comprehension-focused middle 
school reading intervention developed by author and the Strategic Education Research 
Partnership Institute (SERP). STARI is delivered in one teaching block for 3-5 days per week. 
Teachers receive project-authored student workbooks for fluency and comprehension practice, 
unit novels and non-fiction books, slides, and detailed daily lesson plans. In each block, students 
worked for 1 5 minutes on leveled fluency passages, engaging in timed reading, repeated reading, 
phrase-cued reading, and practice with isolated words and phrases. Fluency work was carried out 
in partners and each two-day fluency cycle included a discussion with the partner about a 
controversy or personal response to the fluency passage, as well as checks on literal 
comprehension. 

Comprehension strategies were introduced in teacher mini-lessons and practiced and developed 
through 30-40 minutes daily spent reading program novels and nonfiction books. Work with 
program novels and non-fiction was organized into three thematic units. Students alternated 
reading in small groups, guided by the teacher, and reading in partners, with workbook prompts 
to engage in comprehension strategies such as questioning, summarizing, and predicting. Each 
unit included one or more debates about a controversial issue raised in the unit novels or 
nonfiction. Students gathered information from the unit texts to support positions in the debate 
and prepared debate presentations working in teams. 

Key features of the STARI intervention were extensive reading of text on unit themes of 
importance in students’ lives (bullying, the war in Iraq, diverse families), multiple opportunities 
for students to talk about text, and a focus on developing personal stances on text content. 

Research Design: 

A total of 9 middle schools participated in the study and 562 students were in the core analytic 
sample that was used for the intention-to-treat analyses. With each of the 9 schools, we identified 
eligible students who scored below proficient on the state ELA assessment. Students in 
substantially separate special education classes, students who were level 1 or 2 English language 
learners, and students whose special education plan included intensive, rules-based phonics 
intervention were excluded from study participation. The remaining students were randomized 
into treatment or control conditions. Students assigned to the treatment condition were rank 
ordered as part of the randomization and schools were asked to place students into the available 
seats in intervention classes, following their rank orders. 
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Data Collection and Analysis: 


In this study, we present treatment effects on the Global Integrated Scenarios Based Assessment 
(GISA), using pretest and grade level as covariates, and school fixed effects to account for the 
lottery design. In fall 2013, children were pretested on the GISA, which was developed by a 
team of researchers at ETS (Sabatini, O’Reilly, Halderman, & Bruce, 2014). The GISA is a 45- 
minute, scenario-based assessment of reading comprehension delivered over the Internet. An 
example of a GISA passage is focused on Organic Fanning. Prior to reading the passage, 
students are presented with the following scenario: “Your class has decided to create a website 
about organic farming with the subject. The website will provide information to answer the 
following questions: “What are the natural methods used in organic farming? How are these 
methods different from the methods used on non-organic, or conventional, farms? What are the 
pros and cons of organic farming? You will work with three classmates on the project. Students 
are then asked to read a variety of texts to answer these questions, including texts on “techniques 
used in organic fanning; simulated results of a web search; advantages / disadvantages of organic 
farming; a simulated web discussion; cartoons, charts, and graphic organizers” (Sabatini et al., 
2014, p. 39). 

There are moderately strong conelations between GISA and each of the six subtests in the 
Reading Inventory and Scholastic Evaluation (Sabatini et al., 2013). The six subtests include: 
Word Recognition & Decoding, Vocabulary, Morphological Awareness, Sentence Processing, 
Efficiency of Basic Reading Comprehension, and Reading Comprehension. Sabatini et al. (2014) 
report moderately strong correlations (Pearson’s correlation range between r= .71 and r = .77) 
between the GISA and RISE subtests, providing initial evidence of convergent validity. 

Findings / Results: 

To address our research question, we use ordinary least squares (OLS) regression models to 
identify the causal effect of being randomly assigned via a lottery process to STARI or control 
classrooms. Thus, the ITT estimates offer an unbiased estimate of the treatment effect if 
students were offered the opportunity to participate in the STARI intervention. Using OLS 
regression, we used the following model to generate an unbiased intention-to-treat estimate of 
STARI on the GISA outcome measure: 


(1) E = A> + l>iT, + fcXt + fhRB, + r, 


where Y, represents the spring GISA outcome for student i, T, represents whether the student 
was randomly assigned to STARI, X, is a vector of student background variables (i.e., the pretest 
fall GISA score, grade), RBj represents the school fixed effect, and s, represents the error term. 
The coefficient yS; is the estimated difference in posttest scores between treatment and control 
students and represents the intention-to-treat estimate. 

Overall, the intention-to-treat analysis indicates that STARI students outperfonned control 
students by 5.69 scaled score points (t = 1.91, p = .057). Expressed in standard deviation units, 
the effect size was d = .11. 
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Conclusions: 


This impact analysis extends our prior work by showing that a multi-component Tier-2 
intervention for struggling adolescent readers can improve a range of reading outcomes as well 
as a global, integrated scenario-based assessment of reading. The magnitude of the intention-to- 
treat effect is also comparable to the typical impact of multi-component adolescent literacy 
interventions on norm-referenced reading comprehension tests. However, this is the first 
intervention study of which we are aware that examines the efficacy of a multi-component 
adolescent literacy intervention on a 21 st century literacy measure tapping students’ ability to 
evaluate and integrate infonnation from multiple text sources. Combined with our earlier impact 
study on the RISE, this study adds to the growing research based on effective, multi-component 
adolescent literacy interventions. 
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